Skip to content

Conversation

TomNicholas
Copy link
Member

@TomNicholas TomNicholas commented Sep 10, 2022

Initial attempt to get cubed working within xarray, as an alternative to dask.

I've added a manager kwarg to the .chunk methods so you can do da.chunk(manager="cubed") to convert to a chunked cubed.CoreArray, with the default still being da.chunk(manager="dask"). (I couldn't think of a better name than "manager", as "backend" and "executor" are already taken.)

At the moment it should work except for an import error that I don't understand, see below.

Fro cubed to work at all with this PR we would also need:

To-dos for me on this PR:

  • Re-route xarray.apply_ufunc through cubed.apply_gufunc instead of dask's apply_gufunc when appropriate,
  • Add from_array_kwargs to opening functions, e.g. open_zarr, and open_dataset,
  • Add from_array_kwargs to creation functions, such as full_like,
  • Add store_kwargs as a way to propagate cubed-specific kwargs when saving to_zarr.

To complete this project more generally we should also:

cc @tomwhite

@TomNicholas TomNicholas added topic-arrays related to flexible array support and removed topic-typing labels Sep 10, 2022
@github-actions github-actions bot added topic-typing and removed topic-arrays related to flexible array support labels Sep 10, 2022
Copy link
Contributor

@dcherian dcherian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@TomNicholas
Copy link
Member Author

@Illviljan thanks for all your comments!

Would you (or @keewis?) be willing to approve this PR now? I would really like to merge this so that I can release a version of xarray that I can use as a dependency for cubed-xarray.

@dcherian
Copy link
Contributor

Thanks @TomNicholas Big change!

@dcherian dcherian merged commit 6cd6122 into pydata:main May 18, 2023
@TomNicholas
Copy link
Member Author

Woooo thanks @dcherian !

@jhamman
Copy link
Member

jhamman commented May 18, 2023

👏 Congrats @TomNicholas on getting this in! Such an important contribution. 👏

@TomNicholas TomNicholas mentioned this pull request May 18, 2023
4 tasks
@tomwhite
Copy link
Contributor

Thanks for all your hard work on this @TomNicholas!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API design CI Continuous Integration tools dependencies Pull requests that update a dependency file enhancement io needs review run-benchmark Run the ASV benchmark workflow topic-arrays related to flexible array support topic-backends topic-CF conventions topic-dask topic-indexing topic-rolling topic-typing topic-zarr Related to zarr storage library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Alternative parallel execution frameworks in xarray
9 participants